Systems and methods for rapid electronic messaging testing and positional impact assessment in a prospect electronic messaging series

ABSTRACT

Disclosed are methods, systems, and non-transitory computer-readable medium for optimizing user experience with respect to sequences of electronic messages to drive user engagement. For example, a method may include determining a set of messages including at least a first message and a second message; transmitting the first and second messages as an initial message to a first plurality of devices and a second plurality of devices, respectively; determining a first and second performance indicator at least based on one or more responses to the first and second messages transmitted as the initial message, respectively; comparing the first performance indicator and the second performance indicator; and sorting a sequence of the set of messages based on the comparison of the first performance indicator to the second performance indicator.

TECHNICAL FIELD

Various embodiments of the present disclosure generally relate tooptimizing user experience, and more particularly, to optimizing userexperience with respect to sequences of electronic messages to driveuser engagement.

BACKGROUND

Some businesses utilize multiple series of electronic messages(sometimes referred to as “email journeys”), to drive subscriptions orother engagement (e.g., clicks, conversions, etc.) from prospectivecustomers who have visited an associated website. Email journeys can bethought of as a sequence of emails that are evaluated both individuallyand collectively as a group for effectiveness in increasing yield.Optimizing email journeys to increase yield, e.g., new subscriptions,may be a slow process. Further, it may be difficult to understand howthe position of a particular email in the email journey may impacts itsresults.

Rapid email testing is important to maximize return on investment (ROI)from an email journey with the goal of converting prospective customersinto subscribers. The challenges that prevent rapid email testing for anemail journey include, but are not limited to, the following: (1) therecipient engagement often drops off in a direct correlation to theemail recipient's tenure as a prospect, which results in increased timeto establish a result, e.g., an understanding of the efficacy of theemail series in achieving the desired goal, e.g., new subscriptions, inaggregate; (2) the time it takes to obtain a statistically significanttest, e.g., understanding the efficacy of the email series in achievingthe desired goal is therefore directly correlated to the number ofemails in the series; and (3) difficulty of isolating variables duringthe testing. As an example, the emails themselves may be one variableand the order of the emails may be a second variable, and it may bedifficult to isolate the impact of the two variables. The difficulty ofisolating the impact of particular variables may add further time to thetesting.

Some existing email systems allow testing of an individual email, e.g.,via NB testing, but such testing falls short because the tests areperformed with respect to the individual email and without regard to theconcept of an email journey, i.e., a coordinated series of emailmessages. Accordingly, existing systems do not provide the ability toisolate a positional impact variable. Therefore, there exists a need foremail journey optimization with respect to both the email content foreach email included in the email journey and the sequence of the emailsincluded in the email journey.

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Unless otherwiseindicated herein, the materials described in this section are not priorart to the claims in this application and are not admitted to be priorart, or suggestions of the prior art, by inclusion in this section.

SUMMARY OF THE DISCLOSURE

According to certain aspects of the disclosure, systems and methods aredisclosed for optimizing user experience to overcome the problems withconventional methods noted above.

In one embodiment, a computer-implemented method comprises: determining,by one or more processors, a set of messages including at least a firstmessage and a second message; assigning, by the one or more processors,the first message as a first initial message from the set of messages tobe transmitted to one or more user devices; transmitting, by the one ormore processors, the set of messages to a first plurality of devicesassociated with user identifiers, wherein each of the first plurality ofuser identifiers is associated with a customer of an electronicsubscription platform, and wherein the first message is transmitted asthe first initial message from the set of messages to the firstplurality of devices associated with user identifiers; determining, bythe one or more processors, a first performance indicator at least basedon one or more responses to the first message transmitted as the firstinitial message from the set of messages; assigning, by the one or moreprocessors, the second message as a second initial message from the setof messages to be transmitted to one or more user devices; transmitting,by the one or more processors, the set of messages to a second pluralityof devices associated with user identifiers, wherein each of the secondplurality of user identifiers is associated with a customer of theelectronic subscription platform, and wherein the second message istransmitted as the second initial message from the set of messages tothe second plurality of devices associated with user identifiers;determining, by the one or more processors, a second performanceindicator at least based on one or more responses to the second messagetransmitted as the second initial message from the set of messages;comparing, by the one or more processors, the first performanceindicator and the second performance indicator; and sorting, by the oneor more processors, a sequence of the set of messages based on thecomparison of the first performance indicator to the second performanceindicator.

In accordance with another embodiment, a computer system comprises: adata storage device storing processor-readable instructions; and aprocessor configured to execute the instructions to perform a method.The method may include: determining a set of messages including at leasta first message and a second message; assigning the first message as afirst initial message from the set of messages to be transmitted to oneor more user devices; transmitting the set of messages to a firstplurality of devices associated with user identifiers, wherein each ofthe first plurality of user identifiers is associated with a customer ofan electronic subscription platform, and wherein the first message istransmitted as the first initial message from the set of messages to thefirst plurality of devices associated with user identifiers; determininga first performance indicator at least based on one or more responses tothe first message transmitted as the first initial message from the setof messages; assigning the second message as a second initial messagefrom the set of messages to be transmitted to one or more user devices;transmitting the set of messages to a second plurality of devicesassociated with user identifiers, wherein each of the second pluralityof user identifiers is associated with a customer of the electronicsubscription platform, and wherein the second message is transmitted asthe second initial message from the set of messages to the secondplurality of devices associated with user identifiers; determining asecond performance indicator at least based on one or more responses tothe second message transmitted as the second initial message from theset of messages; comparing the first performance indicator and thesecond performance indicator; and sorting a sequence of the set ofmessages based on the comparison of the first performance indicator tothe second performance indicator.

In accordance with another embodiment, a non-transitorycomputer-readable medium containing instructions that, when executed bya processor, causes the processor to perform a method. The method mayinclude: determining a set of messages including at least a firstmessage and a second message; assigning the first message as a firstinitial message from the set of messages to be transmitted to one ormore user devices; transmitting the set of messages to a first pluralityof devices associated with user identifiers, wherein each of the firstplurality of user identifiers is associated with a customer of anelectronic subscription platform, and wherein the first message istransmitted as the first initial message from the set of messages to thefirst plurality of devices associated with user identifiers; determininga first performance indicator at least based on one or more responses tothe first message transmitted as the first initial message from the setof messages; assigning the second message as a second initial messagefrom the set of messages to be transmitted to one or more user devices;transmitting the set of messages to a second plurality of devicesassociated with user identifiers, wherein each of the second pluralityof user identifiers is associated with a customer of the electronicsubscription platform, and wherein the second message is transmitted asthe second initial message from the set of messages to the secondplurality of devices associated with user identifiers; determining asecond performance indicator at least based on one or more responses tothe second message transmitted as the second initial message from theset of messages; comparing the first performance indicator and thesecond performance indicator; and sorting a sequence of the set ofmessages based on the comparison of the first performance indicator tothe second performance indicator.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate various exemplary embodiments andtogether with the description, serve to explain the principles of thedisclosed embodiments.

FIG. 1 depicts an example environment in which methods, systems, andother aspects of the present disclosure may be implemented.

FIG. 2 depicts a schematic diagram depicting an architecture of anexemplary environment for a clothing-as-a-service electronic platform,according to one or more embodiments.

FIG. 3 depicts an exemplary method of optimizing user experience,according to exemplary embodiments of the present disclosure.

FIG. 4 depicts an exemplary method of optimizing user experience,according to exemplary embodiments of the present disclosure.

FIG. 5 depicts an exemplary computer device or system, in whichembodiments of the present disclosure, or portions thereof, may beimplemented.

DETAILED DESCRIPTION OF EMBODIMENTS

Reference will now be made in detail to the exemplary embodiments of thedisclosure, examples of which are illustrated in the accompanyingdrawings. Wherever possible, the same reference numbers will be usedthroughout the drawings to refer to the same or like parts.

In some embodiments, the present disclosure describes a persistent test,e.g., persistent A/B testing, on the first message of a message series(e.g., that may comprise an email journey) to increase testingthroughput, identify optimal content for each message in the messageseries and determine an optimal order of messages of the series ofmessages. For example, the present disclosure describes a persistenttest for the first email of a message series (also referred to as anemail series) that a prospective customer may receive when the customersigns up as a prospective subscriber of an electronic commerce system,such as a clothing or other product subscription or rental service, forexample. In some embodiments, one or more prospective customers may besubjected to a control version (also referred to as the defaultversion), e.g., an “A” version, or a test version, e.g., a “B” version,of the first email. The A version may be the current implementation of afirst email included in an email journey and the B version may be a testversion of the first email, where the results (e.g., prospectivecustomer responses to the first email) for the B version may be comparedwith the results for the A version. As an example, half of the one ormore prospective customers may receive the A version, while theremaining half of the one or more prospective customers receive the Bversion. In some embodiments, 70 percent of the one or more prospectivecustomers may receive the A version, while the remaining 30 percent ofthe one or more prospective customers receive the B version. Thepercentages of the one or more prospective customers receiving the A andB versions described above are for explanatory purposes only, and mayvary for alternative embodiments.

Each email in the email series may be tested as the B version and theresults may be compared with the results from that of the default Aversion. In some embodiments, each test may be run until statisticalsignificance, e.g., as determined by a 1-tail statistical calculationresulting in a p-value of 0.05 or less, is reached. A performance metricmay be determined for each test for the A and B versions. For example,the clicks per send metric in response to the A and B versions may berecorded and the email with the best clicks per send metric may bedetermined as the optimal email for the first email of the emailjourney.

Once an optimal email is determined, the optimal email may be designatedas the A version and the remaining emails included in the email seriesmay be positioned within the email series based on each respectiveclicks per send metric as measured by respective tests against the Aversion. In some embodiments, each remaining email within the emailseries may be subsequently tested against the A version to determineeach email's respective clicks per send value in an initial ordinalposition test (also referred to as a “boot up processing test”). Theemails within the email series may be ordered based on each respectiveclicks per send results obtained from testing the email in the initialordinal position test. For example, each of the emails included in theemail series may be ordered based on clicks per send in declining order,thereby obtaining an optimally ordered email series. In someembodiments, the optimally ordered email series may be designated as thecurrent email journey.

In some embodiments, the optimally ordered email journey, i.e., thecurrent email journey, may be subsequently evaluated on an individualemail basis based on the same metric, e.g. clicks per send. For example,the clicks per send metric may be determined for an exemplary email “X”which has been reordered for the current email journey, The determinedclicks per send metric for email “X” in its reordered position may becompared to the clicks per send metric for the email “X” in its priorposition, e.g., prior to being reordered in the current email journey.As another example, the determined clicks per send metric for email “X”in its reordered position may be compared to the clicks per send metricfor email “X” when assigned as an initial email, e.g., tested as versionB, during the initial ordinal position test. In such instances, thedifference (i.e., the delta) in the clicks per send metric may beutilized to determine a positional difference which can then be used todetermine the impact of the order based on the clicks per send metric inthe email journey.

In some embodiments, new experimental emails for further inclusion in acurrent email journey ay be designated as the B version, tested, and beincluded in the current email journey based on the test results. Forexample, a new experimental email may be tested as the B version and theresults for the new experimental email may be compared with the resultsfrom that of the default A version (e.g., the initial email for thecurrent email journey). Accordingly, the new experimental email may beincluded in the current email journey based on the comparison. That is,the new experimental email may be positioned in the appropriate positionwithin the current email journey based on the comparison. In someembodiments, the results for the new experimental email may exceed theresults for the default A version. In such embodiments, the newexperimental email may become the default A version for the currentemail journey. In some embodiments, the new experimental email may becompared with other emails included in the current email journey inaddition to the default A version. In such embodiments, the newexperimental email may be included in the current email journey based onthe comparisons. That is, the new experimental email may be placed inthe appropriate order within the current email journey based on thecomparisons.

Current testing platforms do not have tools to facilitate resultprediction based on the positions of emails within an email journey.Accordingly, current testing platforms fail to facilitate ordering ofemails within the email journey to maximize results for the emailjourney.

Significant advantages provided by the current application may include,but are not restricted to the following. First, the current applicationenables a persistent test on the first email of an email series torapidly test email content and rank the order of the emails in the emailseries based on email clicks per recipient. In some embodiments, theordered emails may comprise the current email journey. It is understoodthat the clicks per recipient metric is provided merely as an example,and various metrics may be utilized in alternative embodiments. Second,a position impact curve may be developed based on the use of progressivetesting of the emails within the current email journey via a persistenttest on the first email and a subsequent comparison of the results forindividual emails within the current email journey based on a persistenttest of each individual email in the initial ordinal position of theemail series (e.g., the position of the individual email within theemail series prior to being ordered based on the initial ordinalposition test) compared with the position in the current email journey(e.g., the email series placed in optimal order). The position curve maybe used to predict the clicks per recipient key performance indicator(KPI) of an email based on the test results from the persistent test.This allows the determination of how a new email will perform based on aproposed position. Accordingly, the current application may provide thesignificant advantage of rapidly testing to identify the best emailswithin the email series and optimally sequencing these emails tomaximize a business KPI, e.g., clicks per recipient metric.Additionally, the derived positional result curve may assist inforecasting the results that may be obtained from new content tested inthe initial ordinal position test and placed within the current emailjourney.

Although the present disclosure frequently refers to the use ofelectronic messages in the format known as “e-mail,” it should beappreciated that the present disclosure and the above and belowdescription of “persistent NB testing” are applicable to any type ofelectronic messages, including in-app messages, mobile OS notifications,chat messages, bot messages, SMS messages, or any other type ofelectronic messages that can be sent between users and an electroniccommerce system, such as a clothing subscription service.

In addition, while the exemplary system architecture as described in thepresent disclosure relates to electronic transactions for subscribingto, purchasing, or renting wearable items (e.g., clothing-as-a-service(CaaS) or Try-Then-Buy (TTB) service), implementations disclosed hereinmay effectively serve various other online transactions in the contextof any other subscription, purchase, rental, or retail services withoutdeparting from the scope of the disclosure, such as, for example,subscribing to or making purchases in a software service, cleaningservice, delivery service, maintenance service, rental product, rentalvehicles, etc. In addition, while some descriptions and examplesdisclosed in the present disclosure refer to certain exemplarytransactions as transactions pertaining to “apparel” or “garments,” allof those transactions may effectively serve any wearable item (e.g., anarticle of clothing, apparel, jewelry, hat, accessories, or any otherproduct which may be worn), or even hospitality linens, consumer goods,or any other textile fabrics, without departing from the scope of thedisclosure.

As used in the present disclosure, the term “CaaS” (i.e.,clothing-as-a-service) may collectively refer to computer-implementedservices and functions associated with subscription, purchase, and/orrental services for users (e.g., periodic subscription for receivingwearable items, apparel rental or purchase order, distribution, returnprocessing, TTB services, account management, marketing, customerservice, warehouse operations, etc.). As used in the present disclosure,the term “wearable item” may refer to any article of clothing, apparel,jewelry, hat, accessories, or other product which may be worn by aperson, an animal, or a thing, or be used as an ornament for a person,an animal, or a thing. As used herein, the term “closeting” or “tocloset” may refer to a computer-implemented operation of placing one ormore garments into a virtual closet (e.g., a cart, a repository, or anytype of space which may be virtually associated with a particular set ofone or more garments for a future transaction). Additionally, “matching”may refer to a computer-implemented operation of determining a set ofone or more garments for allocating to a user and/or determiningwearability metrics for given garments, and “allocating” or “allocation”may refer to a computer-implemented operation of determining thegarments that should be assigned and shipped to one or more particularusers.

In accordance with the present disclosure, user interfaces, periodicallyexecuted computer-implemented services, ad hoc services, and automationsbeing integrated together in a connected platform may be achieved by auniquely configured system architecture, job execution clusterconfiguring one or more processors to perform both storefront and backoffice tasks, and various user interfaces providing specialized orcustomized access to users of different roles. For example, the systemmay periodically collect vast amounts of data attributes from historicaltransactions, form data sets indicative of each user's relationship withcertain apparel (e.g., a binary flag of whether a shipped garment wasactually worn by a user) in the back end, and train a neural networkwith those data sets to make specific front-end user recommendationswith highly wearable apparel. The ordered combination of various ad hocand automated tasks in the presently disclosed platform necessarilyachieve technological improvements through the specific processesdescribed more in detail below. In addition, the unconventional andunique aspects of these specific automation processes represent a sharpcontrast to merely providing a well-known or routine environment forperforming a manual or mental task.

The subject matter of the present description will now be described morefully hereinafter with reference to the accompanying drawings, whichform a part thereof, and which show, by way of illustration, specificexemplary embodiments. An embodiment or implementation described hereinas “exemplary” is not to be construed as preferred or advantageous, forexample, over other embodiments or implementations; rather, it isintended to reflect or indicate that the embodiment(s) is/are “example”embodiment(s). Subject matter can be embodied in a variety of differentforms and, therefore, covered or claimed subject matter is intended tobe construed as not being limited to any exemplary embodiments set forthherein; exemplary embodiments are provided merely to be illustrative.Likewise, a reasonably broad scope for claimed or covered subject matteris intended. Among other things, for example, subject matter may beembodied as methods, devices, components, or systems. Accordingly,embodiments may, for example, take the form of hardware, software,firmware, or any combination thereof (other than software per se). Thefollowing detailed description is, therefore, not intended to be takenin a limiting sense.

Throughout the specification and claims, terms may have nuanced meaningssuggested or implied in context beyond an explicitly stated meaning.Likewise, the phrase “in one embodiment” as used herein does notnecessarily refer to the same embodiment and the phrase “in anotherembodiment” as used herein does not necessarily refer to a differentembodiment. It is intended, for example, that claimed subject matterinclude combinations of exemplary embodiments in whole or in part.

The terminology used below may be interpreted in its broadest reasonablemanner, even though it is being used in conjunction with a detaileddescription of certain specific examples of the present disclosure.Indeed, certain terms may even be emphasized below; however, anyterminology intended to be interpreted in any restricted manner will beovertly and specifically defined as such in this Detailed Descriptionsection. Both the foregoing general description and the followingdetailed description are exemplary and explanatory only and are notrestrictive of the features, as claimed.

In this disclosure, the term “based on” means “based at least in parton.” The singular forms “a,” “an,” and “the” include plural referentsunless the context dictates otherwise. The term “exemplary” is used inthe sense of “example” rather than “ideal.” The term “or” is meant to beinclusive and means either, any, several, or all of the listed items.The terms “comprises,” “comprising,” “includes,” “including,” or othervariations thereof, are intended to cover a non-exclusive inclusion suchthat a process, method, or product that comprises a list of elementsdoes not necessarily include only those elements, but may include otherelements not expressly listed or inherent to such a process, method,article, or apparatus. Relative terms, such as, “substantially” and“generally,” are used to indicate a possible variation of ±10% of astated or understood value.

Referring now to the appended drawings, FIG. 1 shows an exampleenvironment 100, according to one or more embodiments of the presentdisclosure. As shown, the example environment 100 may include one ormore networks 101 that interconnect a server system 102, user devices112, employee devices 116, tenant devices 120, and external systems 122.The one or more networks 101 may be, for example, one or more of acellular network, a public land mobile network, a local area network, awide area network, a metropolitan area network, a telephone network, aprivate network, an ad hoc network, an intranet, the Internet, a fiberoptic based network, a cloud computing network, etc. User devices 112may be accessed by users 108, employee devices 116 may be accessed byauthorized employees 114, and tenant devices 120 may be accessed byemployees of tenant entities 118. In some implementations, employeedevices 116 may be used to perform the functions of the tenant devices120 and/or the user devices 112. Server system 102 may comprise one ormore servers 104 and one or more databases 106, which may be configuredto store and/or process a plurality of data, microservices, and servicecomponents, and/or associated functions thereof, as described in moredetail below with respect to FIG. 2.

Users 108 may access the server system 102 through the one or morenetworks 101 using user devices 112. Each device among the user devices112 may be any type of computing device (e.g., personal computingdevice, mobile computing devices, etc.) which allows users 108 todisplay a web browser or a web-based application for accessing theserver system 102 through the network 101. The user devices 112 may, forexample, be configured to display a web browser, a web-basedapplication, or any other user interface (e.g., one or more mobileapplications) for allowing users 108 to exchange information with otherdevice(s) or system(s) in the environment 100 over the one or morenetworks 101. For example, a device among the user devices 110 may loadan application with a graphical user interface (GUI), and theapplication may display on the GUI one or more apparel recommendationsfor closeting by the user. Users 108 accessing user devices 112 may be,for example, users and/or potential users of apparel made available forsubscription-based distribution via electronic transactions and physicalshipment. Additionally, or alternatively, users 108 may access userdevices 112 to, for example, manage one or more user accounts, viewcatalogs, configure one or more user profiles, engage in customerservice communications, make purchase orders, track shipments, generateshipments, monitor order fulfillment processes, initiate or processreturns, order apparel for purchase, provide feedback, refer otherusers, navigate through various features such as size advisor, performpersonalized discovery, and/or make recommendations.

Employee devices 116 may be configured to be accessed by one or moreemployees 114, including, for example, customer service employees,marketer employees, warehouse employees, analytics employees, or anyother employees who are authorized and/or authenticated to performtasks, operations, and/or transactions associated with the server system102, and/or the external systems 122. In one embodiment, employeedevices 116 are owned and operated by the same entity or at least anaffiliate of the entity operating the e-commerce (e.g., CaaS) businesshosted on server systems 102. Each device among the employee devices 116may be any type of computing device (e.g., personal computing device,mobile computing devices, etc.). The employee devices 116 may allowemployees 114 to display a web browser or an application for accessingthe server system 102 and/or the external systems 122, through the oneor more networks 101. For example, a device among the one or more of theemployee devices 116 may load an application with graphical userinterface (GUI), and the application may display on the GUI one or morewarehouse operations associated with providing CaaS to users 108. Insome implementations, the employee devices 116 may communicate directlywith the server system 102 via communications link 117 bypassing publicnetworks 101. Additionally, or alternatively, the employee devices 116may communicate with the server system 102 via network 101 (e.g., accessby web browsers or web-based applications).

Tenant devices 120 may be configured to be accessed by one or moretenants 118. Each device among the tenant devices 120 may be any type ofcomputing device (e.g., personal computing device, mobile computingdevices, etc.). As used herein, each tenant, among one or more tenants118, may refer to an entity that allocates and/or supplies one or morespecific collections of apparel for the CaaS inventory. For example,each of the one or more tenants 118 may be a retailer, a designer, amanufacturer, a merchandizer, or a brand owner entity that supplies oneor more collections of wearable items to the CaaS inventory managedand/or accessed by the server system 102. In some embodiments, tenants118 may use one or more electronic tenant interfaces (e.g., a catalogcontent management system associated with each tenant) to provide theserver system 102 with wearable item data that describe apparel orwearable items made available for electronic transactions on serversystem 102. For example, one or more catalogs for each of the one ormore tenants 118 may be generated and/or updated at the server system102 dynamically and/or periodically. Tenant devices 120 may serve asaccess terminals for the tenants 118, for communicating with theelectronic tenant interfaces and/or other subsystems hosted at theserver system 102. The tenant devices 120 may, for example, beconfigured to display a web browser, an application, or any other userinterface for allowing tenants 118 to load the electronic tenantinterfaces and/or exchange data with other device(s) or system(s) in theenvironment 100 over the one or more networks 101.

External systems 122 may be, for example, one or more third party and/orauxiliary systems that integrate and/or communicate with the serversystem 102 in performing various CaaS tasks. Specific examples of theexternal systems 122 are described in detail below with respect to FIG.2. External systems 122 may be in communication with other device(s) orsystem(s) in the environment 100 over the one or more networks 101. Forexample, external systems 122 may communicate with the server system 102via API (application programming interface) access over the one or morenetworks 101, and also communicate with the employee devices 116 via webbrowser access over the one or more networks 101. In some embodiments,the server system 102 may include one or more components of the externalsystems 122.

As indicated above, FIG. 1 is provided merely as an example. Otherexamples that differ from the example environment 100 of FIG. 1 arecontemplated within the scope of the present embodiments. In addition,the number and arrangement of devices and networks shown in system 100are provided as an example. In practice, there may be additionaldevices, fewer devices and/or networks, different devices and/ornetworks, or differently arranged devices and/or networks than thoseshown in system 100. Furthermore, two or more devices shown in FIG. 1may be implemented within a single device, or a single device shown inFIG. 1 may be implemented as multiple, distributed devices.Additionally, or alternatively, one or more devices may perform one ormore functions of other devices in the example environment 100. Forexample, employee devices 116 may be configured to perform one or morefunctions of tenant devices 120, in addition to their own functions.

FIG. 2 depicts a schematic diagram of an exemplary architecture 200 fora clothing-as-a-service electronic platform, according to one or moreembodiments. The components of the architecture may be accessed byauthorized terminals, such as employee devices 202 and user devices 204,over the one or more networks 101 or via any one or more other types ofnetwork (e.g., a cellular network, a public land mobile network, a localarea network, a wide area network, a metropolitan area network, atelephone network, a private network, an ad hoc network, an intranet,the internet, a fiber optic based network, a cloud computing network,etc.). As used herein, the user devices 204 may correspond to any one ormore user devices 112 depicted in FIG. 1, and the employee devices 202may correspond to any one or more of the employee devices 116 and/ortenant devices 120 depicted in FIG. 1.

As shown in FIG. 2, in general, architecture 200 may comprise aninternal system 206, external consumer facing apps 208, a webapplication hosting server 210, external systems 212, and tenant dataand analytics systems 214. Internal system 206 may comprise internalapps for operations, API endpoints, app endpoints, login endpoints,tenant aware services, tenant aware transactional data stores, backoffice jobs, data warehouse systems, file stores-snapshots, and 3^(rd)party data ETL processes. In some embodiments, the internal system 206may include one or more components of the external systems 212.

In operation, employee devices 202 may access the internal system 206,which may be stored at networked, distributed, and/or local systems(e.g., one or more virtual private clouds and/or one or more physicalprivate networks). As used herein, a virtual private cloud may refer toa configurable pool of shared computing resources within a cloudenvironment, with groups of shared computing resources being allocatedfor a particular job(s), user(s) and/or a purpose(s). The internalsystem 206 may represent a virtual private cloud allocated for hostingthe entirety of the internal system 206. Additionally, the internalsystem 206 may include multiple virtual private clouds (e.g., subsets ofthe larger virtual private cloud encompassing the internal system 206),each allocated for one or more particular tasks, users, or purposes(e.g., tenant aware services, back office jobs, and data warehousesystems).

The employee devices 202 may communicate with the internal system 206through one or more internal apps 216 comprising one or more web-basedportals or user interfaces for operations. The one or more internal apps216 may be, for example, an electronic warehouse operations portaland/or an electronic administrative portal. The one or more internalapps 216 may include one or more web-based user interfaces, such as, forexample, one or more mobile applications compatible with predeterminedmobile device operating systems, a software application developed fordesktop operating systems, and/or a web site configured for browseraccess to web pages via one or more networks (e.g., the Internet or anintranet). Employees 116 or tenants 118 may use employee devices 202 toaccess the internal system 206 and perform various CaaS functionsrelated to warehouse operations and/or administrative tasks.

The internal system 206 may also include Application ProgrammingInterface (“API”) endpoints, APP (application) endpoints, and loginendpoints. In some implementations, these endpoints may be hosted in avirtual private cloud or a physical private network with a preconfiguredlevel of isolation from other virtual private clouds or physical privatenetworks hosting different components of the internal system 206. TheAPI endpoints may be locations from which the APIs can access resourcesof the internal system 206 that the APIs need to carry out theirfunctions. For example, the API endpoints may specify where resourcescan be accessed by the APIs, while communicating with the APIsrequesting information from the internal system 206. Similarly, the APPendpoints may be the locations from which applications (e.g.,applications in one or more internal apps 216) may access resources ofthe internal system 206 they need to carry out their functions, and thelogin endpoints may be the touchpoints of any communication pertainingto logins (e.g., authentication and access control) associated with theinternal system 206. For example, the login endpoints may receive andprocess login communications, for internal apps 216 and/or externalconsumer facing applications 208. For some implementations, at leastsome endpoints among the API endpoints, APP endpoints, and loginendpoints, may be tenant branded endpoints, designated to serveparticular tenants 118.

The internal system 206 may additionally include tenant aware servicesand tenant aware transactional data stores. The tenant aware servicesmay include microservices for fulfillment of various CaaS operationshosted in one or more virtual private clouds or one or more physicalprivate networks. For example, the microservices may include, forexample, catalog data, account data, data services, customer servicefunctions, marketing functions, warehouse functions, and/or othersupporting functions. The tenant aware transactional data stores may beone or more databases that store both raw and processed data resultingfrom operations of the tenant aware services, the endpoints, externalsystems 212, and/or tenant data and analytics systems 214. The tenantaware transactional data stores may store, for example, transactionaldata, batch jobs data, searchable data including various analytics andattributes, event messages, and local logs of various raw data.

The internal system 206 may additionally include back office jobs, whichmay comprise instructions, files, or executable applications configuredto perform various back office tasks and/or computing resources toexecute these instructions or applications. The back office jobs may behosted in, for example, one or more virtual private clouds. The backoffice jobs may include, for example, all system components that run andupdate data (e.g., a first order data or any derived data) associatedwith the internal system 206. Such system components may include, forexample, replenishment identifier (RID) generation service(s), sizeadvisor data set, size advisor algorithmic preparation component(s),recommendation service(s), search data sets, etc. The back office jobsmay include, for example, ETL (extract, transform, and load) processesthat comprise collecting data from multiple different sources (e.g.,tenant aware transactional data stores), converting the collected datato other one or more preset formats as deemed necessary, and loading thedata into the target database (e.g., data warehouse systems). The backoffice jobs may also include, for example, periodic data refreshoperations, periodic synchronization states among internal and externalservices, and automated tasks to run in the background at the internalsystem 206.

The internal system 206 may include the file stores for snapshots in oneor more virtual private clouds or one or more physical private networks.The file stores of snapshots may store snapshots capturing states of theinternal system 206 at particular points in time. For example, eachsnapshot may capture settings, files, configurations, and the memorystate of the components of the internal system 206. These snapshots maybe restored upon request or scheduling, and when a snapshot is restored,settings, and/or the state of the internal system 206 may be returned tothe states they were in at the time the snapshots were captured.

The internal system 206 may include third party data ETL processes,which may collect data from different external sources (e.g., externalsystems 212), convert the collected data to other one or more presetformats as deemed necessary, and load the data into one or more targetdatabases (e.g., data warehouse systems and/or tenant data and analyticssystem 214).

The internal system 206 may include the data warehouse systems 230 inone or more virtual private clouds or one or more physical privatenetworks. As discussed above with respect to the back office jobs andthe third party data ETL processes, the data warehouse systems may bethe one or more target databases for ETL processes that collect datafrom various sources (e.g., the external systems 212 or the tenant awaretransactional data stores). The data warehouse systems may then utilizethe collected data as, for example, parameters for business intelligencethat reveals patterns, analytics, and insights for business decisionsassociated with the internal system 206.

The exemplary environment 200 may include tenant data and analyticssystems 214. The tenant data and analytics systems 214 may be computingresources in communication with one or more components of the internalsystem 206 in order to collect, store, and/or manage data and analyticsassociated with the one or more tenants 118. One or more of the tenantdata and analytics systems 214 may be located remotely from the internalsystem 206 (e.g., at tenant servers). The tenant data and analyticssystems 214 may communicate with the components of the internal system206 using API access or file transfer link 254 over one or more networks101. For example, the tenant data and analytics systems 214 maycommunicate with the API endpoints of the internal system 206, orreceive files from third party data ETL processes. The tenant data andanalytics systems 214 may be accessed by employee devices 202, which maycorrespond to employee devices 116 or the tenant devices 120 depicted inFIG. 1.

As shown in FIG. 2, the exemplary environment 200 may also include userdevices 204, which may correspond to the user devices 112 depicted inFIG. 1. Users 108 of the CaaS electronic platform may use the userdevices 204 to access the internal system 206, as recipients of theservices provided by the components of the internal system 206. Forexample, the users 108 of the user devices 204 may be one or moreregistered subscribers who physically receive and wear the items thatare distributed via the CaaS electronic platform. As another example,the users 108 of the user devices 204 may be one or more potentialsubscribers to the CaaS electronic platform. In the context of thecurrent disclosure, a group of such potential subscribers are referredto as a prospect cohort. As shown in FIG. 2, user devices 204 may accessthe internal system 206 via external consumer facing applications 208.The external consumer facing applications 208 may be browser-accessedweb pages or web-based applications that include web-based userinterfaces accessible from one or more user devices 204 over one or morenetworks (e.g., one or more networks 101).

In some implementations, the exemplary environment 200 may include webapplication hosting server 210 to serve as an intermediary for enablingcommunications made between the user devices 204 and the internal system206. The web application hosting server may be an external (e.g. thirdparty) server that provides an online platform for, for example,building a web-based storefront interface and integrating online retailcomponents (e.g., online point-of-sale system) onto the storefrontinterface, for vendors such as the internal system 206. The webapplication hosting server 210 may communicate with the internal system206 (e.g., the API endpoints, the APP endpoints, and/or an employeedevice 202 logged into the internal system 206), to retrieve necessaryinformation about the internal system 206, and to generate ordynamically update an online storefront for the user devices 204. Theuser devices 204 may, in turn, access the online storefront generated ordynamically updated by the web application hosting server 210, throughthe external consumer facing applications 208. In this way, the externalconsumer facing applications 208 may allow user devices 204 to not onlycommunicate with the internal system 206, but also to communicate withexternal systems 212. Such communication with the external systems 212may be enabled by use of one or more API access communication links 250interconnecting the internal system 206 and the external systems 212.

The exemplary environment 200 may additionally include external systems212 (e.g., systems corresponding to the external systems 122 depicted inFIG. 1), that may be accessed by employee devices 202 (e.g., devicescorresponding to employee devices 116 and/or the tenant devices 120depicted in FIG. 1), and also by the internal system 206 (e.g., thesystem corresponding to server system 102 depicted in FIG. 1). Theemployee devices 202 may access the external systems 212 over one ormore networks, using, for example, web browser access or user interfacesincluded in one or more web-based application. Additionally, as shown inFIG. 2, the internal system 206 may also access the external systemsover the one or more networks, using, for example, one or more APIaccess processes 250.

The external systems 212 may include, for example, analytics systems238, A/B testing systems 240, review systems 244, and email marketingsystems 246. The analytics systems 238 may include one or more webanalytics tools provided by an external (e.g., third party) server, thatprovides dashboards, logs, or reports pertaining to, for example,tracking and reporting website traffic for the vendor (e.g., employees116). For example, an analytics tool may be configured to display poorlyfunctioning pages, where visitors came from, how long the visitorsstayed on the website, the visitors' geographical position, visitorsegmentation information, sales activity and performance, and detailedinformation about current visitors. The analytics systems 238 may beaccessed by employee devices 202 (e.g., a vendor of these solutions) to,for example, customize settings, data, and/or configurations, andutilize the analytics data for business intelligence.

NB testing systems 240 may include NB testing tools for measuring andoptimizing user experience by, for example, measuring subjects' responseto variant A against variant B and determining which of the two variantsis more effective. The NB testing tool may be provided by an external(e.g., third party) server. In the context of the CaaS electronicplatform provided by the internal system 206, the NB testing systems 240may perform an A/B test on, for example, a closet interface with avirtual assistant and a closet interface without a virtual assistant. Asanother example, the A/B testing systems 240 may perform an NB test onemails, as will be described in further detail below. Results of theexperimentations may be provided to employee devices 202 (e.g., a vendorof these solutions) to, for example, utilize the results for businessintelligence.

The review systems 244 may include user review receiving tools providedby an external (e.g., third party) server. For example, a user reviewreceiving tool in the review systems 244 may provide a platform forusers to add reviews, ratings, and/or user generated content such asvideos, to be published for display at the external consumer facingapplications 208. The review systems 244 may be accessed by employeedevices 202 (e.g., a vendor of these tools) to, for example, importreviews for analytics and business intelligence, and/or customizesettings and configurations.

The email marketing systems 246 may include email marketing automationand analytics tools, provided by an external (e.g., third party) server.For example, an email marketing automation tool may maintain mailinglists and mailing schedules, and may modify email marketing messagesbased on what recipients read, click on, or forward. The email marketingsystems 246 may be accessed by employee devices 202 (e.g., a vendor ofthese tools) to, for example, manage communication preferences, thecontent, and/or vendor subscription settings.

The number and arrangement of devices, components, and communicationnetworks shown in FIG. 2 are provided as an example. In practice, theremay be additional devices, components, and/or communication networks,fewer devices, components, and/or communication networks, differentdevices, components, and/or communication networks, or differentlyarranged devices, components, and/or communication networks than thoseshown in FIG. 2. Furthermore, two or more devices shown in FIG. 2 may beimplemented within a single device, or a single device shown in FIG. 2may be implemented as multiple, distributed devices. Additionally, oralternatively, a set of devices (e.g., one or more devices) ofenvironment 200 may perform one or more functions described as beingperformed by another set of devices of environment 200.

FIG. 3 depicts an exemplary method 300 of optimizing user experience,according to exemplary embodiments of the present disclosure. In oneembodiment, method 300 may be performed by one or more combinations ofinternal systems 206, internal apps for operations 216, employee devices202, as well as external systems 212, including analytics systems 236,NB testing systems 240, review systems 244, and e-mail marketing systems246, to perform a method of testing and sorting electronic messages,such as e-mails, to drive user engagement and subscriptions of anelectronic commerce platform, such as a clothing rental service.Specifically, any of the systems, devices, and components of FIG. 2 mayinteract to perform the steps of method 300 to send messages to users(prospective customers or subscribers), analyze and test responses tothose messages, sort and revise messages, and send additional or newmessages to prospective customers or subscribers. It is understood thatthe steps depicted in and described with reference to FIG. 3 may beperformed in any order, e.g., in parallel.

As shown in FIG. 3, in step 302, a set of messages including at least afirst message and a second message may be determined. In someembodiments, the first and second messages may contain any type ofcontent that may provide value to a prospective subscriber. For example,the first and second messages may each be one of the following: amonetary value email (e.g., a rent value email), a funnel email, anoffer email, a recommendation email, a time sensitive offer email, arecommendation email or a testimonial email. In some embodiments, thefirst message and the second message may be positioned in an initialorder. For example, the initial order may indicate that the firstmessage may be transmitted before the second message. As anotherexample, the first message may be the A version of a first emailincluded an email journey and the second message may be the B version asdescribed herein. In step 304, the first message may be assigned as afirst initial message from the set of messages to be transmitted to oneor more user devices. In step 306, the set of messages may betransmitted to a first plurality of devices associated with useridentifiers, where the first message is transmitted as the first initialmessage from the set of messages to the first plurality of devicesassociated with user identifiers. In some embodiments, each of the firstplurality of user identifiers may be associated with a customer of anelectronic subscription platform. In step 308, a first performanceindicator may be determined at least based on one or more responses tothe first message transmitted as the first initial message from the setof messages. In some embodiments, determining the first performanceindicator may include determining a first click per send measurement forthe customers associated with the first plurality of user identifiers.

In step 310, the second message may be assigned as a second initialmessage from the set of messages to be transmitted to one or more userdevices. In step 312, the set of messages may be transmitted to a secondplurality of devices associated with user identifiers, where the secondmessage may be transmitted as the second initial message from the set ofmessages to the second plurality of devices associated with useridentifiers. In some embodiments, each of the second plurality of useridentifiers may be associated with a customer of the electronicsubscription platform. In step 314, a second performance indicator maybe determined at least based on one or more responses to the secondmessage transmitted as the second initial message from the set ofmessages. In some embodiments, determining the second performanceindicator includes determining a second click per send measurement forthe customers associated with the second plurality of user identifiers.

In some embodiments, the first performance indicator and/or the secondperformance indicator may be compared with a predetermined threshold.For example, the determined first click per send measurement and/or thedetermined second click per send measurement may be compared with apredetermined click per send measurement. In such embodiments, the firstmessage and/or the second message may be discarded upon a determinationthat the first performance indicator and/or the second performanceindicator does not exceed the predetermined threshold. It is understoodthat the comparison of the first performance indicator and the secondperformance indicator with the predetermined threshold may be performedin any order. For example, the comparison of the first performanceindicator and the second performance indicator with the predeterminedthreshold may be performed in parallel.

In step 316, the first performance indicator may be compared with thesecond performance indicator. In some embodiments, comparing the firstperformance indicator and the second performance indicator may includecomparing the determined first click per send measurement and thedetermined second click per send measurement. In step 318, a sequence ofthe set of messages may be sorted based on the comparison of the firstperformance indicator to the second performance indicator. In someembodiments, sorting the sequence of the set of messages based on thecomparison of the first performance indicator to the second performanceindicator may include ordering the first and second messages based onthe comparison of the first performance indicator to the secondperformance indicator.

The sorted sequence of the set of messages (hereinafter referred to asthe sorted set of messages) may indicate an updated order of the firstand second messages, thereby positioning the first and second messageswithin the set of messages based on the updated order. As an example,the updated order of the first and second messages may be the same asthe initial order of the first and second messages. As another example,the updated order of the first and second messages may be a reversal ofthe initial order of the first and second messages. In some embodiments,the sorted set of messages may be transmitted to a plurality of devicesassociated with user identifiers. In such embodiments, the first andsecond messages may be transmitted in accordance to the updated order.In some embodiments, an updated first performance indicator may bedetermined at least based on responses to the first message in theupdated position within the sorted set of messages. The updated firstperformance indicator may be compared with the first performanceindicator to determine a positional value of the first message in theupdated position within the sorted set of messages. Similarly, anupdated second performance indicator may be determined at least based onresponses to the second message in the updated position within thesorted set of messages. The updated second performance indicator may becompared with the second performance indicator to determine a positionalvalue of the second message in the updated position within the sortedset of messages.

In some embodiments, the method 300 may include an additional step oftransmitting the set of messages, e.g., the first and second messages,to a plurality of devices associated with user identifiers in accordanceto the initial order. In such embodiments, an initial first performanceindicator may be determined at least based on responses to the firstmessage in the initial position within the set of messages. Similarly,an initial second performance indicator may be determined at least basedon responses to the second message in the initial position within theset of messages.

In some embodiments, the updated first performance indicator may becompared with the initial first performance indicator to determine apositional value of the first message in the updated position within thesorted set of messages. Similarly, the updated second performanceindicator may be compared with the initial second performance indicatorto determine a positional value of the second message in the updatedposition within the sorted set of messages.

FIG. 4 depicts an exemplary method 400 of optimizing user experience,according to exemplary embodiments of the present disclosure. It isunderstood that the steps depicted in and described with reference toFIG. 4 may be performed in any order, e.g., in parallel. In step 402, aset of messages including at least a first message, a second message,and a third message may be determined. In some embodiments, the first,second, and third messages may contain any type of content that mayprovide value to a potential subscriber. For example, the first, second,and third message may each be one of the following: a monetary valueemail (e.g., a rent value email), a funnel email, an offer email, arecommendation email, a time sensitive offer email, a recommendationemail or a testimonial email. The method 400 next proceeds to steps 304through 314 as depicted in and described with reference FIG. 3.

In step 404, the third message may be assigned as a third initialmessage from the set of messages to be transmitted to one or more userdevices. In step 406, the set of messages may be transmitted to a thirdplurality of devices associated with user identifiers, where the thirdmessage is transmitted as the third initial message from the set ofmessages to the third plurality of devices associated with useridentifiers. In some embodiments, each of the third plurality of useridentifiers may be associated with a customer of an electronicsubscription platform. In step 408, a third performance indicator may bedetermined at least based on one or more responses to the third messagetransmitted as the third initial message from the set of messages. Insome embodiments, determining the third performance indicator mayinclude determining a third click per send measurement for the customersassociated with the third plurality of user identifiers.

In some embodiments, the first performance indicator, the secondperformance indicator, and/or the third performance indicator may becompared with a predetermined threshold. For example, the determinedfirst click per send measurement, the determined second click per sendmeasurement, and/or the determined third click per send measurement maybe compared with a predetermined click per send measurement. In suchembodiments, the first message, the second message, and/or the thirdmessage may be discarded upon a determination that the first performanceindicator, the second performance indicator, and/or the thirdperformance indicator does not exceed the predetermined threshold. It isunderstood that the comparison of the first performance indicator, thesecond performance indicator, and third performance indicator with thepredetermined threshold may be performed in any order. For example, thecomparison of the first performance indicator, the second performanceindicator, and the third performance indicator with the predeterminedthreshold may be performed in parallel.

In step 410, the first performance indicator, the second performanceindicator, and the third performance indicator may be compared. In someembodiments, comparing the first performance indicator, the secondperformance indicator, and the third performance indicator may includecomparing the determined first click per send measurement, thedetermined second click per send measurement, and the determined thirdclick per send measurement. In step 412, the sequence of the set ofmessages may be sorted based on the comparison of the first performanceindicator, the second performance indicator, and the third performanceindicator. In some embodiments, sorting the sequence of the set ofmessages based on the comparison of the first performance indicator, thesecond performance indicator, and the third performance indicator mayinclude ordering the first, second, and third messages based on thecomparison of the first performance indicator, the second performanceindicator, and the third performance indicator.

As shown in FIG. 5, a device 500 used for performing the variousembodiments of the present disclosure (e.g., the server system 102, theuser devices 112, the employee devices 116, the tenant devices 120,and/or any other computer system or user terminal for performing thevarious embodiments of the present disclosure) may include a centralprocessing unit (CPU) 520. CPU 520 may be any type of processor deviceincluding, for example, any type of special purpose or a general-purposemicroprocessor device. As will be appreciated by persons skilled in therelevant art, CPU 520 also may be a single processor in amulti-core/multiprocessor system, such system operating alone, or in acluster of computing devices operating in a cluster or server farm. CPU520 may be connected to a data communication infrastructure 510, forexample, a bus, message queue, network, or multi-core message-passingscheme.

A device 500 (e.g., the server system 102, the user devices 112, theemployee devices 116, the tenant devices 120, and/or any other computersystem or user terminal for performing the various embodiments of thepresent disclosure) may also include a main memory 540, for example,random access memory (RAM), and may also include a secondary memory 530.Secondary memory, e.g., a read-only memory (ROM), may be, for example, ahard disk drive or a removable storage drive. Such a removable storagedrive may comprise, for example, a floppy disk drive, a magnetic tapedrive, an optical disk drive, a flash memory, or the like. The removablestorage drive in this example reads from and/or writes to a removablestorage unit in a well-known manner. The removable storage unit maycomprise a floppy disk, magnetic tape, optical disk, etc., which is readby and written to by the removable storage drive. As will be appreciatedby persons skilled in the relevant art, such a removable storage unitgenerally includes a computer usable storage medium having storedtherein computer software and/or data.

In alternative implementations, secondary memory 530 may include othersimilar means for allowing computer programs or other instructions to beloaded into device 500. Examples of such means may include a programcartridge and cartridge interface (such as that found in video gamedevices), a removable memory chip (such as an EPROM, or PROM) andassociated socket, and other removable storage units and interfaces,which allow software and data to be transferred from a removable storageunit to device 500.

A device 500 may also include a communications interface (“COM”) 560.Communications interface 560 allows software and data to be transferredbetween device 500 and external devices. Communications interface 560may include a modem, a network interface (such as an Ethernet card), acommunications port, a PCMCIA slot and card, or the like. Software anddata transferred via communications interface may be in the form ofsignals, which may be electronic, electromagnetic, optical, or othersignals capable of being received by communications interface 560. Thesesignals may be provided to communications interface 560 via acommunications path of device 500, which may be implemented using, forexample, wire or cable, fiber optics, a phone line, a cellular phonelink, an RF link or other communications channels.

The hardware elements, operating systems, and programming languages ofsuch equipment are conventional in nature, and it is presumed that thoseskilled in the art are adequately familiar therewith. A device 500 alsomay include input and output ports 550 to connect with input and outputdevices such as keyboards, mice, touchscreens, monitors, displays, etc.Of course, the various server functions may be implemented in adistributed fashion on a number of similar platforms, to distribute theprocessing load. Alternatively, the servers may be implemented byappropriate programming of one computer hardware platform.

The systems, apparatuses, devices, and methods disclosed herein aredescribed in detail by way of examples and with reference to thefigures. The examples discussed herein are examples only and areprovided to assist in the explanation of the apparatuses, devices,systems, and methods described herein. None of the features orcomponents shown in the drawings or discussed below should be taken asmandatory for any specific implementation of any of these theapparatuses, devices, systems, or methods unless specifically designatedas mandatory. For ease of reading and clarity, certain components,modules, or methods may be described solely in connection with aspecific figure. In this disclosure, any identification of specifictechniques, arrangements, etc. are either related to a specific examplepresented or are merely a general description of such a technique,arrangement, etc. Identifications of specific details or examples arenot intended to be, and should not be, construed as mandatory orlimiting unless specifically designated as such. Any failure tospecifically describe a combination or sub-combination of componentsshould not be understood as an indication that any combination orsub-combination is not possible. It will be appreciated thatmodifications to disclosed and described examples, arrangements,configurations, components, elements, apparatuses, devices, systems,methods, etc. can be made and may be desired for a specific application.Also, for any methods described, regardless of whether the method isdescribed in conjunction with a flow diagram, it should be understoodthat unless otherwise specified or required by context, any explicit orimplicit ordering of steps performed in the execution of a method doesnot imply that those steps must be performed in the order presented butinstead may be performed in a different order or in parallel.

Throughout this disclosure, references to components or modulesgenerally refer to items that logically can be grouped together toperform a function or group of related functions. Like referencenumerals are generally intended to refer to the same or similarcomponents. Components and modules can be implemented in software,hardware, or a combination of software and hardware. The term “software”is used expansively to include not only executable code, for examplemachine-executable or machine-interpretable instructions, but also datastructures, data stores and computing instructions stored in anysuitable electronic format, including firmware, and embedded software.The terms “information” and “data” are used expansively and includes awide variety of electronic information, including executable code;content such as text, video data, and audio data, among others; andvarious codes or flags. The terms “information,” “data,” and “content”are sometimes used interchangeably when permitted by context.

It is intended that the specification and examples be considered asexemplary only, with a true scope and spirit of the disclosure beingindicated by the following claims.

1-20. (canceled)
 21. A computer-implemented method comprising:transmitting, by one or more processors, a set of messages to aplurality of devices associated with a plurality of user identifiers,according to an initial order, the set of messages including a firstinitial message and a second initial message; determining, by the one ormore processors, a first performance indicator based on one or moreresponses to the first initial message in the set of messages;determining, by the one or more processors, a second performanceindicator based on the one or more responses to the second initialmessage in the set of messages; comparing, by the one or moreprocessors, the first performance indicator and the second performanceindicator with a predetermined threshold; updating, by the one or moreprocessors, an order of the set of messages based on the comparing; andtransmitting, by the one or more processors, the set of messagesaccording to the updated order.
 22. The computer-implemented method ofclaim 21, wherein determining the first performance indicator includesdetermining a first click per send measurement for the plurality ofdevices.
 23. The computer-implemented method of claim 22, whereindetermining the second performance indicator includes determining asecond click per send measurement for the plurality of devices.
 24. Thecomputer-implemented method of claim 23, wherein the comparing includescomparing the first click per send measurement and the second click persend measurement with a predetermined click per send measurement. 25.The computer-implemented method of claim 21, the method furthercomprising: in response to determining that the first performanceindicator does not exceed the predetermined threshold, discarding, bythe one or more processors, the first initial message.
 26. Thecomputer-implemented method of claim 21, the method further comprising:in response to determining that the second performance indicator doesnot exceed the predetermined threshold, discarding, by the one or moreprocessors, the second initial message.
 27. The computer-implementedmethod of claim 21, the updated order of the set of messages indicatinga reversed order of the first initial message and the second initialmessage.
 28. The computer-implemented method of claim 21, wherein theupdated order is identical to the initial order of the first initialmessage and the second initial message.
 29. A computer systemcomprising: a data storage device storing processor-readableinstructions; and a processor configured to execute instructions toperform a method including: transmitting a set of messages to aplurality of devices associated with a plurality of user identifiers,according to an initial order, the set of messages including a firstinitial message and a second initial message; determining a firstperformance indicator based on one or more responses to the firstinitial message in the set of messages; determining a second performanceindicator based on the one or more responses to the second initialmessage in the set of messages; comparing the first performanceindicator and the second performance indicator with a predeterminedthreshold; updating an order of the set of messages based on thecomparing; and transmitting the set of messages according to the updatedorder.
 30. The computer system of claim 29, wherein determining thefirst performance indicator includes determining a first click per sendmeasurement for the plurality of devices.
 31. The computer system ofclaim 30, wherein determining the second performance indicator includesdetermining a second click per send measurement for the plurality ofdevices.
 32. The computer system of claim 31, wherein the comparingincludes comparing the first click per send measurement and the secondclick per send measurement with a predetermined click per sendmeasurement.
 33. The computer system of claim 29, the method furthercomprising: in response to determining that the first performanceindicator does not exceed the predetermined threshold, discarding thefirst initial message.
 34. The computer system of claim 29, the methodfurther comprising: in response to determining that the secondperformance indicator does not exceed the predetermined threshold,discarding the second initial message.
 35. The computer system of claim29, the order of the set of messages indicating a reversed order of thefirst initial message and the second initial message.
 36. The computersystem of claim 29, wherein the initial order is identical to theinitial order of the first initial message and the second initialmessage.
 37. A non-transitory computer-readable medium containinginstructions that, when executed by a processor, cause the processor toperform a method comprising: transmitting, by one or more processors, aset of messages to a plurality of devices associated with a plurality ofuser identifiers, according to an initial order, the set of messagesincluding a first initial message and a second initial message;determining, by the one or more processors, a first performanceindicator based on one or more responses to the first initial message inthe set of messages; determining, by the one or more processors, asecond performance indicator based on the one or more responses to thesecond initial message in the set of messages; comparing, by the one ormore processors, the first performance indicator and the secondperformance indicator with a predetermined threshold; updating, by theone or more processors, an order of the set of messages based on thecomparing; and transmitting, by the one or more processors, the set ofmessages according to the updated order.
 38. The non-transitorycomputer-readable medium of claim 37, wherein determining the firstperformance indicator includes determining a first click per sendmeasurement for the plurality of devices.
 39. The non-transitorycomputer-readable medium of claim 38, wherein determining the secondperformance indicator includes determining a second click per sendmeasurement for the plurality of devices.
 40. The non-transitorycomputer-readable medium of claim 39, wherein the comparing includescomparing the first click per send measurement and the second click persend measurement with a predetermined click per send measurement.